---

title: Unwindowed datasets


keywords: fastai
sidebar: home_sidebar

summary: "This functionality will allow you to create a dataset that applies sliding windows to the input data on the fly. This heavily reduces the size of the input data files, as only the original, unwindowed data needs to be stored."
description: "This functionality will allow you to create a dataset that applies sliding windows to the input data on the fly. This heavily reduces the size of the input data files, as only the original, unwindowed data needs to be stored."
nb_path: "nbs/014_data.unwindowed.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/014_data.unwindowed.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<ul>
<li>I'd like to thank both Thomas Capelle (<a href="https://github.com/tcapelle">https://github.com/tcapelle</a>)  and Xander Dunn (<a href="https://github.com/xanderdunn">https://github.com/xanderdunn</a>) for their contributions to make this code possible. </li>
</ul>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="TSUnwindowedDataset" class="doc_header"><code>class</code> <code>TSUnwindowedDataset</code><a href="https://github.com/timeseriesAI/tsai/tree/main/tsai/data/unwindowed.py#L12" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>TSUnwindowedDataset</code>(<strong><code>X</code></strong>, <strong><code>y</code></strong>=<em><code>None</code></em>, <strong><code>y_func</code></strong>=<em><code>None</code></em>, <strong><code>window_size</code></strong>=<em><code>1</code></em>, <strong><code>stride</code></strong>=<em><code>1</code></em>, <strong><code>drop_start</code></strong>=<em><code>0</code></em>, <strong><code>drop_end</code></strong>=<em><code>0</code></em>, <strong><code>seq_first</code></strong>=<em><code>True</code></em>, <strong>**<code>kwargs</code></strong>)</p>
</blockquote>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h2 id="TSUnwindowedDatasets" class="doc_header"><code>class</code> <code>TSUnwindowedDatasets</code><a href="https://github.com/timeseriesAI/tsai/tree/main/tsai/data/unwindowed.py#L69" class="source_link" style="float:right">[source]</a></h2><blockquote><p><code>TSUnwindowedDatasets</code>(<strong><code>dataset</code></strong>, <strong><code>splits</code></strong>) :: <code>FilteredBase</code></p>
</blockquote>
<p>Base class for lists with subsets</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">y_func</span><span class="p">(</span><span class="n">y</span><span class="p">):</span> <span class="k">return</span> <span class="n">y</span><span class="o">.</span><span class="n">astype</span><span class="p">(</span><span class="s1">&#39;float&#39;</span><span class="p">)</span><span class="o">.</span><span class="n">mean</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>This approach works with both univariate and multivariate data.</p>
<ul>
<li>Univariate: we'll use a simple array with 20 values, one with the seq_len first (X0), the other with seq_len second (X1).</li>
<li>Multivariate: we'll use 2 time series arrays, one with the seq_len first (X2), the other with seq_len second (X3). No sliding window has been applied to them yet. </li>
</ul>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">X0</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span>
<span class="n">X1</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span> <span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">X0</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">X0</span><span class="p">,</span> <span class="n">X1</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">X1</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>((20,),
 array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,
        17, 18, 19]),
 (1, 20),
 array([[ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15,
         16, 17, 18, 19]]))</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">X2</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">100</span><span class="p">])</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span>
<span class="n">X3</span> <span class="o">=</span> <span class="n">np</span><span class="o">.</span><span class="n">arange</span><span class="p">(</span><span class="mi">20</span><span class="p">)</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="mi">1</span><span class="p">,</span><span class="o">-</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="n">np</span><span class="o">.</span><span class="n">array</span><span class="p">([</span><span class="mi">1</span><span class="p">,</span> <span class="mi">10</span><span class="p">,</span> <span class="mi">100</span><span class="p">])</span><span class="o">.</span><span class="n">reshape</span><span class="p">(</span><span class="o">-</span><span class="mi">1</span><span class="p">,</span><span class="mi">1</span><span class="p">)</span>
<span class="n">X2</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">X3</span><span class="o">.</span><span class="n">shape</span><span class="p">,</span> <span class="n">X2</span><span class="p">,</span> <span class="n">X3</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>((20, 3),
 (3, 20),
 array([[   0,    0,    0],
        [   1,   10,  100],
        [   2,   20,  200],
        [   3,   30,  300],
        [   4,   40,  400],
        [   5,   50,  500],
        [   6,   60,  600],
        [   7,   70,  700],
        [   8,   80,  800],
        [   9,   90,  900],
        [  10,  100, 1000],
        [  11,  110, 1100],
        [  12,  120, 1200],
        [  13,  130, 1300],
        [  14,  140, 1400],
        [  15,  150, 1500],
        [  16,  160, 1600],
        [  17,  170, 1700],
        [  18,  180, 1800],
        [  19,  190, 1900]]),
 array([[   0,    1,    2,    3,    4,    5,    6,    7,    8,    9,   10,
           11,   12,   13,   14,   15,   16,   17,   18,   19],
        [   0,   10,   20,   30,   40,   50,   60,   70,   80,   90,  100,
          110,  120,  130,  140,  150,  160,  170,  180,  190],
        [   0,  100,  200,  300,  400,  500,  600,  700,  800,  900, 1000,
         1100, 1200, 1300, 1400, 1500, 1600, 1700, 1800, 1900]]))</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

<div class="cell border-box-sizing text_cell rendered"><div class="inner_cell">
<div class="text_cell_render border-box-sizing rendered_html">
<p>Now, instead of applying SlidingWindow to create and save the time series that can be consumed by a time series model, we can use a dataset that creates the data on the fly. In this way we avoid the need to create and save large files. This approach is also useful when you want to test different sliding window sizes, as otherwise you would need to create files for every size you want to test.The dataset will create the samples correctly formatted and ready to be passed on to a time series architecture.</p>

</div>
</div>
</div>
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">wds0</span> <span class="o">=</span> <span class="n">TSUnwindowedDataset</span><span class="p">(</span><span class="n">X0</span><span class="p">,</span> <span class="n">window_size</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">stride</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">seq_first</span><span class="o">=</span><span class="kc">True</span><span class="p">)[:][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">wds1</span> <span class="o">=</span> <span class="n">TSUnwindowedDataset</span><span class="p">(</span><span class="n">X1</span><span class="p">,</span> <span class="n">window_size</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">stride</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">seq_first</span><span class="o">=</span><span class="kc">False</span><span class="p">)[:][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">test_eq</span><span class="p">(</span><span class="n">wds0</span><span class="p">,</span> <span class="n">wds1</span><span class="p">)</span>
<span class="n">wds0</span><span class="p">,</span> <span class="n">wds0</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">wds1</span><span class="p">,</span> <span class="n">wds1</span><span class="o">.</span><span class="n">data</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>(TSTensor(samples:8, vars:1, len:5),
 tensor([[[ 0,  1,  2,  3,  4]],
 
         [[ 2,  3,  4,  5,  6]],
 
         [[ 4,  5,  6,  7,  8]],
 
         [[ 6,  7,  8,  9, 10]],
 
         [[ 8,  9, 10, 11, 12]],
 
         [[10, 11, 12, 13, 14]],
 
         [[12, 13, 14, 15, 16]],
 
         [[14, 15, 16, 17, 18]]]),
 TSTensor(samples:8, vars:1, len:5),
 tensor([[[ 0,  1,  2,  3,  4]],
 
         [[ 2,  3,  4,  5,  6]],
 
         [[ 4,  5,  6,  7,  8]],
 
         [[ 6,  7,  8,  9, 10]],
 
         [[ 8,  9, 10, 11, 12]],
 
         [[10, 11, 12, 13, 14]],
 
         [[12, 13, 14, 15, 16]],
 
         [[14, 15, 16, 17, 18]]]))</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="n">wds2</span> <span class="o">=</span> <span class="n">TSUnwindowedDataset</span><span class="p">(</span><span class="n">X2</span><span class="p">,</span> <span class="n">window_size</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">stride</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">seq_first</span><span class="o">=</span><span class="kc">True</span><span class="p">)[:][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">wds3</span> <span class="o">=</span> <span class="n">TSUnwindowedDataset</span><span class="p">(</span><span class="n">X3</span><span class="p">,</span> <span class="n">window_size</span><span class="o">=</span><span class="mi">5</span><span class="p">,</span> <span class="n">stride</span><span class="o">=</span><span class="mi">2</span><span class="p">,</span> <span class="n">seq_first</span><span class="o">=</span><span class="kc">False</span><span class="p">)[:][</span><span class="mi">0</span><span class="p">]</span>
<span class="n">test_eq</span><span class="p">(</span><span class="n">wds2</span><span class="p">,</span> <span class="n">wds3</span><span class="p">)</span>
<span class="n">wds2</span><span class="p">,</span> <span class="n">wds3</span><span class="p">,</span> <span class="n">wds2</span><span class="o">.</span><span class="n">data</span><span class="p">,</span> <span class="n">wds3</span><span class="o">.</span><span class="n">data</span>
</pre></div>

    </div>
</div>
</div>

<div class="output_wrapper">
<div class="output">

<div class="output_area">



<div class="output_text output_subarea output_execute_result">
<pre>(TSTensor(samples:8, vars:3, len:5),
 TSTensor(samples:8, vars:3, len:5),
 tensor([[[   0,    1,    2,    3,    4],
          [   0,   10,   20,   30,   40],
          [   0,  100,  200,  300,  400]],
 
         [[   2,    3,    4,    5,    6],
          [  20,   30,   40,   50,   60],
          [ 200,  300,  400,  500,  600]],
 
         [[   4,    5,    6,    7,    8],
          [  40,   50,   60,   70,   80],
          [ 400,  500,  600,  700,  800]],
 
         [[   6,    7,    8,    9,   10],
          [  60,   70,   80,   90,  100],
          [ 600,  700,  800,  900, 1000]],
 
         [[   8,    9,   10,   11,   12],
          [  80,   90,  100,  110,  120],
          [ 800,  900, 1000, 1100, 1200]],
 
         [[  10,   11,   12,   13,   14],
          [ 100,  110,  120,  130,  140],
          [1000, 1100, 1200, 1300, 1400]],
 
         [[  12,   13,   14,   15,   16],
          [ 120,  130,  140,  150,  160],
          [1200, 1300, 1400, 1500, 1600]],
 
         [[  14,   15,   16,   17,   18],
          [ 140,  150,  160,  170,  180],
          [1400, 1500, 1600, 1700, 1800]]]),
 tensor([[[   0,    1,    2,    3,    4],
          [   0,   10,   20,   30,   40],
          [   0,  100,  200,  300,  400]],
 
         [[   2,    3,    4,    5,    6],
          [  20,   30,   40,   50,   60],
          [ 200,  300,  400,  500,  600]],
 
         [[   4,    5,    6,    7,    8],
          [  40,   50,   60,   70,   80],
          [ 400,  500,  600,  700,  800]],
 
         [[   6,    7,    8,    9,   10],
          [  60,   70,   80,   90,  100],
          [ 600,  700,  800,  900, 1000]],
 
         [[   8,    9,   10,   11,   12],
          [  80,   90,  100,  110,  120],
          [ 800,  900, 1000, 1100, 1200]],
 
         [[  10,   11,   12,   13,   14],
          [ 100,  110,  120,  130,  140],
          [1000, 1100, 1200, 1300, 1400]],
 
         [[  12,   13,   14,   15,   16],
          [ 120,  130,  140,  150,  160],
          [1200, 1300, 1400, 1500, 1600]],
 
         [[  14,   15,   16,   17,   18],
          [ 140,  150,  160,  170,  180],
          [1400, 1500, 1600, 1700, 1800]]]))</pre>
</div>

</div>

</div>
</div>

</div>
    {% endraw %}

</div>
 

